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TRANSFORMATION  OF  ARMY  TEST  AND  EVALUATION 


The  purpose  of  Test  and  Evaluation  (T&E)  during  the  development  and  acquisition  of  a 
defense  system  is  to  identify  and  understand  the  areas  of  risk  that  must  be  accepted,  reduced , 
or  eliminated.1  A  desire  to  control  costs  and  to  reduce  acquisition  cycle  time  (and  more  quickly 
field  the  latest  technological  advances)  often  is  manifested  in  pressure  to  reduce  the  scope  or 
forego  portions  of  the  T&E  process,  including  Operational  Test  and  Evaluation  (OT&E). 

However,  as  a  1 987  GAO  report  on  OT&E  recognizes ,  inadequate  T&E  can  lead  to  increased 
operational  risk: 

If  adequate  OT&E  is  not  done  and  the  weapon  system  does  not  perform 
satisfactorily  in  the  field,  significant  changes  may  be  required.  Moreover,  the 
changes  will  not  be  limited  to  a  few  developmental  models,  but  may  also  be 
applied  to  items  already  produced  and  deployed.  In  extreme  situations,  DoD  also 
risks  (1 )  deploying  systems,  which  cannot  adequately  perform  significant  portions 
of  their  missions,  thus  degrading  our  deterrent/defensive  capabilities  and  (2) 
endangering  the  safety  of  military  personnel  who  operate  and  maintain  the 
systems.2 

While  the  primary  purpose  of  T&E  is  to  support  acquisition  decisions,  a  secondary 
purpose  is  to  support  the  broader  Army  and  Defense  “analysis  community”.  This  community  is 
comprised  of  the  various  analysis  organizations  that  determine  force  structure  and  warfighting 
requirements.  Test  data  are  used  to  validate  the  estimates  of  systems  performance  and  the 
Models  and  Simulations  (M&S)  that  are  used  by  these  organizations.  Similarly,  validation  with 
test  data  improves  the  pedigree  of  the  performance  estimates  that  are  used  by  operational 
forces  in  exercises  and  experiments  as  well  as  in  weaponeering.  Weaponeering  is  defined  by 
the  Air  Force  as  “the  process  of  estimating  the  quantity  of  a  specific  type  weapon  required  to 
achieve  a  specific  level  of  damage  to  a  given  target,  considering  target  vulnerability,  weapon 
effects,  munition  delivery  errors,  damage  criteria,  probability  of  kill,  weapon  reliability,  etc.”3 

During  the  past  decade,  the  Army  has  implemented  major  changes  to  the  organizational 
structure,  responsibilities ,  and  relationships  in  its  T&E  and  analysis  organizations  .  These 
changes  were  intended  to  improve  efficiency  by  consolidating  similar  functions  that  were 
previously  scattered  across  various  Army  organizations.  While  these  changes  have  led  to 
improvements ,  room  for  future  improvement  still  exists.  For  example,  T&E  functions  are  still  split 
between  the  Army  Test  and  Evaluation  Command  (ATEC)  and  the  Army  Materiel  Command 
(AMC).  Furthermore,  the  changes  made  to  date  have  not  eliminated  the  long-running  criticism 
that  the  T&E  process  does  not  adequately  test  systems  in  a  realistic  joint  operational  context. 


This  Strategy  Research  Project  describes  the  strategic  relevance  of  T&E,  characterizes  the 
interaction  of  the  Army’s  analysis  community  with  T&E,  and  reviews  the  organizational  evolution 
of  the  Army's  T&E  infrastructure.  It  then  proposes  additional  changes  to  streamline  business 
processes  and  improve  Army  evaluation  and  analysis  capabilities. 

STRATEGIC  RELEVANCE 

The  Research,  Development,  Test  and  Evaluation  (RDT&E)  of  military  systems  directly 
supports  the  Ends ,  Ways,  and  Means  that  are  set  forth  in  various  defense  policy  documents. 

For  example,  the  Defense  Policy  Goals  {Ends)  contained  in  the 2001  Quadrennial  Defense 
Revieware  “Assuring  Allies  and  Friends,”  “Dissuading  Future  Military  Competition,”  “Deterring 
Threats  and  Coercion  Against  U.S.  Interests,”  and  “If  Deterrence  Fails,  Decisively  Defeat  Any 
Adversary.”  Among  the  seven  Strategic  Tenets  ( Ways)  to  achieve  these  goals,  RDT&E  is  a 
component  of  four:  “Managing  Risks”,  “A  Capabilities-Based  Approach”,  “Developing  a  Broad 
Portfolio  of  Military  Capabilities”,  and  “Transforming  Defense”.4  The  Quadrennial  Defense 
Review  also  points  out  that  one  of  the  four  “pillars”  of  transformation  is  experimentation  using 
wargaming,  simulations  and  field  exercises. 

The  objective  (End)  of  the  Defense  Acquisition  System  (of  which  RDT&E  is  primary 
amongst  the  Ways),  is  “to  acquire  quality  products  that  satisfy  user  needs  with  measurable 
improvements  to  mission  capability  and  operational  support,  in  a  timely  manner,  and  at  a  fair 
and  reasonable  price.”5  The  standard  T&E  process  (Means)  currently  used  to  support  the 
acquisition  of  new  materiel  is  described  in  the  DoD  Interim  Defense  Acquisition  Guidebook  and 
Army  Regulation  73-1 ,  Test  and  Evaluation  Policy.  The  “product”  of  the  Army  T&E  process  is 
an  understanding  of  system  capabilities  ,  which  is  documented  in  integrated  (developmental  and 
operational)  evaluations  used  to  inform  production  and  fielding  decisions.  This  process  consists 
of  the  collection  of  data  from  Developmental  Tests,  Operational  Tests,  Modeling  and  Simulation 
(M&S),  Demonstrations ,  and  Experiments  in  order  to  evaluate  the  Effectiveness,  Suitability,  and 
Survivability  of  the  system  under  development.6  Developmental  Test  and  Evaluation  is  used  as 
an  engineering  development/design  tool  and  to  verify  the  inherent  technical  capabilities  offered 
by  new  technologies  and  systems.7  Operational  Test  and  Evaluation  is  defined  as: 

the  field  test,  under  realistic  combat  conditions ,  of  any  item  of  (or  key  component 
of)  weapons,  equipment,  or  munitions  for  the  purpose  of  determining  the 
effectiveness  and  suitability  of  the  weapons,  equipment  or  munitions  for  use  in 
combat  by  typical  military  users;  and  the  evaluation  of  the  results  of  such  test.8 
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Operational  Test  and  Evaluation  is  thus  used  to  assess  the  degree  to  which  soldiers  can 
leverage  a  system’s  technical  capabilities  in  a  realistic  operational  field  test  context.  Title  10, 
United  States  Code  sets  forth  statutory  requirements  for  the  conduct  of  Operational  Test  and 
Evaluation9  and  realistic  survivability/lethality  Live  Fire  Test  and  Evaluation10  for  major  systems 
and  munitions  prior  to  proceeding  beyond  Low-Rate  Initial  Production. 

T&E  AND  ANALYSIS  ORGANIZATION  INTERDEPENDENCIES 

Numerous  organizations  within  the  Army  and  Defense  Department  rely  on  weapon- 
system  data  in  the  conduct  of  their  analysis  missions.  Figure  1  depicts  the  typical  information 
flow  between  these  organizations.  The  paragraphs  that  follow  describe  the  missions  and 
relationships  of  key  organizations  in  Army  T&E  and  analysis. 


ATEC 


Test  Results 


Information 


Forces 


FIGURE  1 .  T&E  AND  ANALYSIS  INTERDEPENDENCIES 

DIRECTOR,  OPERATIONAL  TEST  AND  EVALUATION  (DOT&E) 

In  1 983,  Congress  created  the  statutory  requirement  for  the  Defense  Department  to 
establish  a  DOT&E  to  oversee  Service  Operational  Test  and  Evaluation  programs.  In  1994,  the 
DOT&E  assumed  responsibility  for  oversight  of  Live  Fire  Test  and  Evaluation  (LFT&E) 
programs.  The  Director  of  DOT&E,  who  is  a  Presidential  appointee  requiring  Senate 
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confirmation,  prepares  annual  reports  to  Congress  on  ail  major  defense  acquisition  programs. 
The  functions  and  duties  of  DOT&E  are  described  in  Title  X,  United  States  Code,  Sections  139, 
2366,  2399  and  2400."  The  responsibilities  of  DOT&E  are  specified  in  Section  139: 


The  Director  shall - 

(1)  prescribe,  by  authority  of  the  Secretary  of  Defense,  policies  and  procedures 
for  the  conduct  of  operational  test  and  evaluation  in  the  Department  of  Defense; 

(2)  provide  guidance  to  and  consult  with  the  Secretary  of  Defense  and  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  and  the 
Secretaries  of  the  military  departments  with  respect  to  operational  test  and 
evaluation  in  the  Department  of  Defense  in  general  and  with  respect  to  specific 
operational  test  and  evaluation  to  be  conducted  in  connection  with  a  major 
defense  acquisition  program; 

(3)  monitor  and  review  all  operational  test  and  evaluation  in  the  Department  of 
Defense; 

(4)  coordinate  operational  testing  conducted  jointly  by  more  than  one  military 
department  or  defense  agency; 

(5)  review  and  make  recommendations  to  the  Secretary  of  Defense  on  all 
budgetary  and  financial  matters  relating  to  operational  test  and  evaluation, 
including  operational  test  facilities  and  equipment,  in  the  Department  of  Defense; 
and 

(6)  monitor  and  review  the  live  fire  testing  activities  of  the  Department  of  Defense 
provided  for  under  section  2366  of  this  title.12 

DOT&E  is  also  responsible  for  the  execution  of  the  Joint  Technical  Coordinating  Group  for 
Munitions  Effectiveness  (JTCG/ME),  which  has  the  mission  of  publishing  operational 
effectiveness  estimates  for  all  non-nuclear  weapons  and  standardizing  effectiveness  measures 
and  methodologies  used  by  the  services.  These  estimates  are  published  in  Joint  Munitions 
Effectiveness  Manuals  and  include  descriptions  of  weapon  system  characteristics  ,  such  as 
detection  ranges,  engagement  ranges,  fly-out  times,  delivery  accuracy,  reliability,  and  kill 
probabilities.  JTCG/ME  data  and  Joint  Munitions  Effectiveness  Manuals  are  used  by  a  variety  of 
U.S.  and  Allied  combatant  commanders,  analysis  organizations,  trainers  ,  and  defense  planners 
in  weaponeering,  training,  tactics  development  and  weapons  systems  studies.13 

ARMY  TEST  AND  EVALUATION  COMMAND  (ATEC) 

The  Army  is  unique  among  the  services  in  having  a  single  organization,  ATEC,  which  is 
responsible  for  developmental  testing,  operational  testing,  and  the  continuous  (through  all 
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phases  of  a  program's  life  cycle)  integrated  (developmental  and  operational)  evaluation  of 
materiel.  The  primary  “products”  of  ATEC  are  test  data  and  systems  evaluations.  DOT&E  uses 
ATEC  products  as  the  primary  input  for  “beyond  low-rate  initial  production”  reports  to  Congress . 
Program  Managers  and  Acquisition  Executives  also  use  ATEC  products  in  materiel  acquisition 
and  fielding  decisions.  ATEC  evaluations  determine  the  degree  to  which  materiel  is  effective, 
suitable,  and  survivable.  These  evaluations  typically  include  recommended  system 
improvements  or  “operational  work-arounds”  (changes  to  Tactics,  Techniques  and  Procedures) 
to  overcome  capability  limitations  observed  in  testing. 

The  ATEC  commander  is  a  major  general  who  reports  directly  to  the  Vice  Chief  of  Staff  of 
the  Army.  ATEC  is  comprised  of  three  major  subordinate  commands.  The  Developmental  Test 
Command,  headquartered  at  Aberdeen  Proving  Ground,  Maryland,  manages  developmental 
test  centers  throughout  the  U  .S.  and  plans,  conducts ,  and  reports  on  developmental  tests.  The 
Operational  Test  Command,  headquartered  at  Fort  Hood,  Texas,  manages  operational  test 
centers  throughout  the  U  .S.  and  plans,  conducts ,  and  reports  on  operational  tests.  The  Army 
Evaluation  Center,  headquartered  in  Alexandria,  Virginia,  develops  evaluation  plans,  determines 
data  requirements  and  sources  (analysis,  developmental  testing,  operational  testing,  M&S, 
exercises),  observes  testing,  and  evaluates  system  effectiveness,  suitability,  and  survivability. 
Also  unique  among  the  services  is  the  fact  that  ATEC,  as  the  Army’s  Operational  Test  Agency  is 
responsible  for  defining  LFT&E  requirements  and  reporting  on  LFT&E  results  (program 
managers  assume  this  responsibility  in  other  services).  The  unique  characteristics  of  ATEC 
activities  were  endorsed  by  a  1999  Defense  Science  Board  recommendation ,  which  implicitly 
urged  the  other  services  to  adopt  the  Army/ATEC  model: 

Each  of  the  Service  DT&OT  organizations  should  be  consolidated,  to  include 
integrated  planning,  use  of  models,  simulation  and  data  reduction.  Planning 
should  be  totally  integrated,  and  the  OSD  T&E  organizations  consolidated.  There 
should  be  integrated  use  of  models,  simulation  and  data  reduction.  Except  for 
limited  dedicated  Operational  Test  and  Evaluation  (OT&E),  contractor  and 
government  testing  should  also  be  integrated.14 

ARMY  RESEARCH  LABORATORY  (ARL) 

ARL  is  a  component  of  the  Army  Materiel  Command’s  Research  Development  and 
Engineering  Command.  Two  elements  of  ARL  are  heavily  involved  in  the  T&E  process  -  the 
Survivability/Lethality  Analysis  Directorate  and  the  Human  Research  and  Engineering 
Directorate. 
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Survivability/Lethality  Analysis  Directorate  (SLAD) 

SLAD’s  authorized  workforce  consists  of  one  hundred  and  forty  seven  employees  at 
Aberdeen  Proving  Ground,  Maryland,  one  hundred  and  seventeen  employees  at  White  Sands 
Missile  Range  and  twenty  employees  at  Fort  Monmouth,  New  Jersey.  The  mission  of  SLAD  is 
to: 


-  Provide  survivability  lethality,  and  vulnerability  (SLV)  analysis  and  evaluation 
support  over  the  entire  life  cycle  of  major  Army  systems  and  help  acquire 
systems  that  will  survive  and/or  be  highly  lethal  in  all  environments  against 
the  full  spectrum  of  battlefield  threats. 

Provide  advice/consultation  on  SLV  issues  to  HQDA,  PEOs/PM,  evaluators, 
combat  developers,  battle  labs,  intelligence  activities,  and  other  DA  and  DOD 
activities. 

-  Conduct  investigations,  experiments,  simulations,  and  analyses  to  quantify 
SLV  of  Army  and  selected  foreign  weapon  systems. 

Provide  well-documented  timely  technical  judg ments  on  complex  SLV  issues. 

-  Perform  special  studies  and  make  recommendations  regarding  tactics, 
techniques,  or  design  modifications  to  reduce  vulnerability  and  enhance 
survivability  and  lethality  of  Army  materiel. 

-  Develop  tools,  techniques,  and  methodologies  for  improving  SLV  analysis.15 

ATEC  relies  on  SLAD  analyses,  methodology,  M&S,  and  testing  for  input  to  the 
effectiveness  and  survivability  portions  of  ATEC  evaluations.  SLAD  conducts  all  modeling  and 
simulation  that  supports  the  pre-test  predictions  and  leads  the  damage  assessment  teams  for  all 
Army  Live  Fire  Test  and  Evaluation  tests  SLAD  also  manages  and  conducts  live  fire  testing  of 
Army  aviation  systems.  AMSAA  relies  on  SLAD  vulnerability  modeling  as  input  in  its  weapon 
systems  effectiveness  calculations.  SLAD  relies  on  results  of  ATEC  conducted  tests  to  validate 
the  penetration  and  behind-armor  debris  algorithms  of  SLAD  vulnerability  models. 

Human  Research  and  Engineering  Directorate  (HRED) 

Headquartered  at  Aberdeen  Proving  Ground,  Maryland,  HRED  has  an  authorized  staff  of 
two  hundred  and  thirty-five  employees,  whose  mission  is: 


To  optimize  soldier  effectiveness  and  soldier-machine  interactions  and  to  ensure 
that  future  system  designs  will  enable  our  soldiers  to  achieve  maximum 
performance.  To  fulfill  this  mission,  HRED  conducts  broad-based  scientific 
research  and  technology  application  and  provides  leadership  in  human  factors 
integration  and  support  to  MANPRINT  (manpower  and  personnel  integration).16 
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ATEC  relies  on  HRED  MANPRINT  analyses  in  the  evaluation  of  weapons  systems 
suitability. 

ARMY  MATERIEL  SYSTEMS  ANALYSIS  ACTIVITY  (AMSAA) 

AMSAA  is  also  an  element  in  the  Army  Materiel  Command's  Research  Development  and 
Engineering  Command.  The  mission  of  AMSAA  is  to  “support  the  Army  decision-making 
process  by  providing  materiel,  logistics  and  industrial  systems  analyses,  as  well  as  business 
process,  manpower  and  resource  analysis.” 17  AMSAA  headquarters  and  the  majority  of 
AMSAA’s  workforce  of  three  hundred  and  eight  employees  is  located  at  Aberdeen  Proving 
Ground,  Maryland.  Small  satellite  offices  are  located  in  Philadelphia,  Pennsylvania;  Letterkenny, 
Pennsylvania;  Redstone  Arsenal,  Alabama;  and  Rock  Island,  Illinois. 

AMSAA’s  core  business  areas  include  methodology  development;  analysis;  and  M&S 
development,  use,  verification  and  validation.18  AMSAA  provides  and  certifies  item-level  and 
system-level  performance  data  for  use  in  all  Army  studies.  The  JTCG/ME  Coordination  Office  in 
AMSAA  manages  the  JTCG/ME  program  for  DOT&E.  Item  and  system  level  performance  data 
that  are  provided  by  AMSAA  or  are  contained  in  Joint  Munitions  Effectiveness  Manuals  include 
signatures,  accuracy,  fly-out  times,  hit  and  kill  probabilities,  reliability,  engagement  time-lines, 
emplacement/displacement  times,  and  transit  times.  ATEC  uses  AMSAA  item-level 
performance  estimates  in  the  real-time  casualty  assessment  instrumentation  that  is  used  at 
ATEC  test  ranges.  ATEC  relies  on  AMSAA  analytical  support  in  the  areas  of  weapons  systems 
effectiveness  (such  as  using  SLAD  vulnerability/lethality  estimates  and  test-based  delivery 
accuracy  to  determine  kill  probabilities),  suitability  (such  as  wholesale/retail  supply  and 
maintenance  analyses),  and  survivability  (such  as  missile  in-flight  survivability).  AMSAA  relies 
on  information  from  ATEC  tests  and  evaluations  to  update  their  item-level  and  system-level 
performance  databases. 

PROGRAM  MANAGER  (PM) 

The  PM  is  responsible  for  managing  all  aspects  of  a  given  materiel  acquisition  program 
and  ensuring  that  program  requirements,  including  T&E,  are  achieved  within  approved  resource 
and  time  constraints.  The  PM,  working  with  his  prime  contractor,  develops  system  specifications 
that  are  intended  to  result  in  the  development  of  a  system  that  meets  TRADOC  developed 
operational  requirements.  Until  validated  or  updated  by  test  results,  these  requirements  and 
specifications  typically  form  the  basis  for  item  level  performance  data  and  Model  and  Simulation 
inputs. 
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TRAINING  AND  DOCTRINE  COMMAND  (TRADOC) 

The  principal  analysis  activity  within  TRADOC  is  the  TRADOC  Analysis  Center  (TRAC), 
located  at  Fort  Leavenworth,  Kansas  and  White  Sands  Missile  Range,  New  Mexico.  TRAC  is 
the  Army  agency  responsible  for  conducting  the  studies  that  form  the  basis  of  new  operations 
and  organization  (O&O)  concepts  and  weapon  systems  operational  requirements.  TRAC  is  also 
the  Army’s  lead  analysis  agency  for  the  conduct  of  Advanced  Warfighting  Experiments.  TRAC 
develops  and  uses  various  models  and  simulations  (M&S)  in  performing  its  mission.19  These 
M&S  require  as  input  “item  level  performance  data”  that  characterize  weapons  systems. 
TRADOC  combat  developers  generate  the  Tactics,  Techniques  and  Procedures  as  well  as  the 
Critical  Operational  Issues  and  Criteria  that  are  used  by  ATEC  in  the  design,  conduct,  and 
evaluation  of  Operational  Tests. 

CENTER  FOR  ARMY  ANALYSIS  (CAA) 

CAA,  located  in  Fort  Belvoir,  Virginia,  is  the  Army's  center  of  excellence  for 
theater/campaign  warfare  analysis.  The  CAA  mission: 

CAA  is  an  analysis  organization  that  supports  FIQDA  and  Major  Army 
Commands.  CAA  develops  information  that  helps  Army  top  management 
address  the  issues  of  greatest  importance  to  the  Army.  CAA  develops 
information  by  conducting  studies  employing  analysis  techniques  appropriate  to 
the  issues  at  hand.  CAA  maintains  special  expertise  in  the  analysis  of  issues 
pertaining  to  theater-level  operations  and  Army-wide  processes,  especially  those 
involving  resource  allocation.”20 

Every  two  years,  CAA  conducts  the  quantitative  analysis  portion  of  the  “Total  Army 
Analysis”  to  determine  unconstrained  force  structure  requirements.  The  Concepts  Evaluation 
Model ,  used  by  CAA  to  determine  battle  movement,  personnel  and  equipment  losses, 
consumption  rates,  etc.,  relies  on  friendly  and  enemy  weapons  effectiveness  data  amongst  its 
inputs.21  These  weapons  effectiveness  data  are  provided  by  AMSAA. 

OPERATIONAL  FORCES 

Operational  Forces  include  the  Combatant  Commands,  Component  Commands,  and 
unattached  U.S.  Army  Forces  Command  units.  These  units  typically  rely  on  Joint  Munitions 
Effectiveness  Manuals  or  AMSAA  effectiveness  data  for  the  weaponeering  component  of 
mission  planning  and  in  the  real-time  casualty  assessment  instrumentation  systems  that  are 
used  in  training  and  exercises.  ATEC  and  Project  Managers  receive  post-deployment 
information  from  operational  forces  for  use  in  continuous  evaluations. 
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RECENT  ORGANIZATIONAL  CHANGES 

ATEC  is  a  fairly  new  organization ;  it  was  created  at  the  culmination  of  a  series  of  major 
organizational  and  procedural  changes  in  Army  T&E  that  took  place  in  the  1990’s.  In  1996, 
AMSAA  conducted  a  study  of  T&E  reengineering  as  part  of  the  Functional  Area  Assessment  of 
Equip,  Supply,  and  Maintain  functions  .  This  study  concluded  that  significant  efficiencies  would 
be  achieved  through  the  integration  of  developmental  and  operational  evaluation.22  As  a  result 
of  this  study,  Army  Evaluation  was  transformed  in  October  1 996,  when  the  Developmental 
Evaluation  responsibilities  and  resources  of  the  Army  Materiel  Command  were  assigned  to  the 
Evaluation  Analysis  Center,  a  new  organization  within  the  Army’s  Operational  Test  and 
Evaluation  Command  (OPTEC).  The  workforce  of  the  Evaluation  Analysis  Center  was  created 
by  reassigning  ninety-eight  employees  from  AMSAA,  eighteen  from  SLAD,  and  thirty-five  from 
the  Army  Materiel  Command's  Test  and  Evaluation  Command. 

In  October  1 999  the  Army  Test  and  Evaluation  Command  (ATEC)  was  created  to 
consolidate  Army  T&E.  ATEC  was  created  by  reassigning  and  redesignating  the  technical  test 
responsibilities  and  resources  of  AMC's  Test  and  Evaluation  Command  as  ATEC’s 
Developmental  Test  Command  and  by  redesignating  OPTEC’s  Test  and  Experiment  Command 
as  the  ATEC’s  Operational  Test  Command.  Also  as  part  of  this  consolidation,  evaluation  was 
further  transformed  by  merging  the  technical  evaluation  responsibilities  of  OPTEC’s  Evaluation 
Analysis  Center  with  the  operational  evaluation  responsibilities  of  OPTEC’s  Operational 
Evaluation  Command  to  create  ATEC’s  Army  Evaluation  Center  (AEC),  which  now  has  an 
authorized  workforce  of  five  hundred  and  eighteen  employees.23 

The  Research,  Development  and  Engineering  Command  (RDECOM)  of  AMC  was 
provisionally  established  in  October  2002.  Figure  2  depicts  the  RDECOM  organizational 
structure.  RDECOM  consists  of  the  research,  development  and  engineering  elements  that  had 
previously  been  assigned  to  the  various  AMC  Major  Subordinate  Commands.  For  example,  the 
Tank  and  Automotive  Research,  Development  and  Engineering  Command  (TARDEC)  had 
previously  been  a  part  of  the  Tank  and  Automotive  Command  (TACOM).  The  mission  of 
RDECOM  is  to  “Field  technologies  which  sustain  America’s  Army  as  the  premier  land  force  in 
the  world.’24 
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FIGURE  2.  RDECOM  ORGANIZATION2526 


ROOM  FOR  IMPROVEMENT 

The  major  criticisms  of  the  current  T&E  process  relate  to  cycle  time,  costs,  and  test 
realism  -  the  degree  to  which  the  use  of  equipment  in  tests  is  representative  of  the  way  we 
fight,  particularly  in  view  of  the  joint  nature  of  current  and  future  operations.  But  the  old  adage 
“fast,  cheap,  good  -  you  cannot  have  all  three”  applies.  As  Michael  Wynn,  Principal 
Undersecretary  of  Defense  for  Acquisition,  testified: 

In  fact,  as  we  proceed  through  trying  to  shorten  the  cycle  of  development  to  get 
this  technology  to  our  war  fighters,  we  would  anticipate  that  we  would  encounter 
more  risk,  not  less  risk,  and  that  therefore  we  would  be  encountering  more  test 
failures,  not  less  test  failures.27 

The  job  of  T&E  planners  is  to  attempt  to  balance  the  conflicting  objectives  of  speed, 
affordability,  realism  and  risk  in  their  developmental  and  operational  test  designs.  Indicators  of 
the  importance  of  realism  in  T&E  are  the  facts  that  in  the  period  from  1996-2000,  system 
reliability  requirements  were  met  in  only  20%  of  operational  tests,  and  operator  error  accounted 
for  approximately  50%  of  system  failures.28  Therefore,  realistic  “players”  conducting 
operationally  realistic  tasks  under  realistic  conditions  are  essential  to  assess  a  system’s 
operational  reliability. 

To  expedite  materiel  fielding  in  critical  cases,  the  Army  uses  an  Urgent  Materiel  Release 
process  to  quickly  field  equipment  that  is  urgently  needed  (in  specific  quantity,  duration,  and 
location)  in  support  of  specific  operations.  This  process  requires  that  the  gaining  command 
accept,  at  the  General  Officer  level,  all  known  equipment  and  supportability  issues  and  risks, 
including  safety  and  health  hazards,  operational  limitations  and  restrictions  of  use.  The  Urgent 
Materiel  Release  process  is  to  be  used  only  in  cases  where  a  deployed  or  deploying  force  has 
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an  operational  need  that  cannot  otherwise  be  satisfied  from  Army  or  Defense  inventories  . 
However,  this  process  does  not  eliminate  the  need  to  eventually  conduct  sufficient  data 
collection  to  adequately  evaluate  the  system.29 

One  source  of  inefficiency  in  the  planning  of  T&E  is  the  number  of  organizations  that  are 
involved  in  the  process.  One  of  the  findings  in  the  1999  Defense  Science  Board  Taskforce  on 
Test  and  Evaluation  was  that  “Bureaucratic  barriers  to  cooperation  and  efficiencies  are 
contributors  to  an  increasingly  protracted  weapons  system  development  process.”30  T&E 
overhead  is  unnecessarily  high  because  each  organization  typically  sends  representatives  to 
numerous  T&E  planning  meetings,  even  though  their  missions  are  closely  related.  Furthermore, 
the  approval  of  T&E  planning  and  evaluation  documents  is  encumbered  by  the  need  for 
management  review  of  the  input  from  each  organization.  Program  Managers  and  DOT&E  action 
officers  are  frequently  confused  because  they  are  forced  to  negotiate  support,  resources,  and 
time-lines  with  representatives  from  multiple  organizations  (for  example,  ATEC,  SLAD  and 
AMSAA  each  have  a  role  in  many  LFT&E  programs).  This  bureaucratic  structure  complicates 
the  prioritization  and  allocation  of  limited  resources. 

Questionable  operational  realism  has  been  a  long-running  criticism  of  operational  testing. 
In  1 989,  the  Assistant  Comptroller  General  testified  that  “in  our  1 983  report  on  DOD's  joint 
operational  test  and  evaluation,  we  found  that  unrealistic  test  conditions...  raised  serious 
questions  about  the  validity  of  the  evaluations  conducted  jointly  by  the  services.”31  Operational 
realism  in  a  joint  environment  is  an  issue  of  such  significance  that  the  DoD  Transformation 
Planning  Guidance  requires: 

Transformation  of  Test  and  Evaluation:  As  the  Department  transforms  to  a  joint 
concept-centric  approach  to  operational  planning  and  capabilities  development, 
we  need  integrated  architectures  that  define  the  specific  parameters  of  the 
requisite  joint  capabilities.  A  Joint  Test  and  Evaluation  Capability  (JointTEC)  is 
needed  to  test  the  capabilities  in  a  realistic  joint  environment  ...  Test  and 
evaluation  in  a  joint  context  will  reveal  whether  or  not  the  integrated  architectures 
present  a  viable  application  of  warfighting  capabilities.  A  JointTEC  would  focus 
policies,  plans,  methodologies,  and  resources  for  evaluation  in  joint  operations 
environments.32 

ARMY  STRUCTURAL  SUGGESTIONS 

Efficiencies  could  be  gained  in  T&E  planning  and  execution  by  combining  AEC,  SLAD, 
HRED  and  AMSAA  into  a  new  AEC  -  a  unified  Analysis  &  Evaluation  Center  under  ATEC.  This 
consolidation  would  build  upon  the  T&E  restructuring  that  was  conducted  in  the  1990’s  and 
would  be  consistent  with  the  recommendations  of  the  1999  Defense  Science  Board  Task  Force 
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on  Test  and  Evaluation.  This  consolidation  would  also  address  a  Government  Accounting  Office 
criticism  that  separate,  independent  reviews  of  T&E  facilities  and  research  and  development 
laboratories  in  the  1 995  Base  Realignment  and  Closure  process  created  “artificial  barriers”  to 
infrastructure  reduction.33 

Such  consolidation  would  provide  “unity  of  command”  in  T&E  planning  and  execution ;  it 
would  eliminate  redundant  management  (including  at  least  two  Senior  Executive  Service  level 
and  numerous  GS-1 4/1 5  level  positions)  and  staff  support  structures.  This  consolidation  under 
ATEC  would  provide  greater  command  emphasis  to  the  work  currently  performed  by  SLAD, 
AMSAA  and  HRED  by  reducing  the  number  of  command  levels  between  the  action  officers  and 
the  Department  of  Army  customer. 

An  example  of  the  lack  of  Research,  Development  and  Engineering  Command  emphasis 
on  the  work  of  these  organizations  is  the  fact  that  of  the  1 97  articles  in  the  first  ten  issues  of 
RDECOM  magazine  (covering  1 1  months),  only  one  cited  work  that  was  being  done  at  AMSAA, 
SLAD  or  HRED.34  This  ratio  of  one  half  of  one  percent  is  significantly  less  than  we  would  expect 
from  almost  six  percent  of  the  Research,  Development  and  Engineering  Command  workforce. 
This  disparity  suggests  that  these  analytical  elements  do  not  receive  as  much  command 
attention  (and  hence  priority)  as  do  the  elements  that  develop  technologies,  which  is  the  main 
mission  of  the  Research,  Development  and  Engineering  Command. 

Similarly,  consolidation  would  streamline  the  planning  and  execution  of  JTCG/ME  tasks 
and  would  eliminate  organizational  barriers  that  inhibit  timely  collection  and  inclusion  of  test  data 
into  Joint  Munitions  Effectiveness  Manuals.  Additional  benefits  of  consolidation  would  be  the 
elimination  of  the  requirement  for  Project  Managers  to  negotiate,  coordinate  and  fund  T&E 
support  from  multiple  organizations  and  the  creation  of  “one-stop-shopping”  for  DOT&E 
interaction  with  the  Army  on  T&E  and  JTCG/ME  issues. 

Opponents  of  this  consolidation  might  argue  against  it  in  two  principal  ways  .  First, 
opponents  might  claim  that  this  consolidation  would  dilute  the  existing  AEC  mission  by  noting 
that  there  are  aspects  of  the  SLAD,  HRED  and  AMSAA  missions  that  are  not  strictly  related  to 
evaluation.  Considering  the  “continuous  evaluation”  requirements  of  the  AEC  mission,  the 
SLAD  non-evaluation  workload  is,  in  reality,  very  small,  consisting  mainly  of  its  test  and 
experiment  range  capabilities.  In  the  proposed  consolidation,  these  capabilities  would  logically 
be  integrated  into  ATEC’s  Developmental  Test  Command’s  facilities  at  Aberdeen  Proving 
Ground  and  White  Sands  Missile  Range. 

Similar  counter-arguments  would  apply  for  HRED  as  well.  AMSAA's  item/system-level 
performance,  acquisition  support,  M&S,  and  significant  portions  of  its  logistics  work  efforts  are 
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directly  applicable  to  continuous  evaluation.  However,  some  of  its  logistics  work  efforts  (such  as 
war  reserve  and  contingency  force  package  development)  and  its  business  and  resource 
analysis  work  efforts  are  not  directly  related  to  AEC’s  evaluation  mission.  To  avoid  dilution  of  the 
AEC  mission,  these  tasks  and  the  associated  resources  could  easily  be  divested  to  another 
organization  within  the  Army  Materiel  Command,  such  as  the  Logistics  Evaluation  Agency. 

Second,  opponents  might  argue  that  because  of  the  extent  of  support  that  SLAD,  HRED 
and  AMSAA  provide  directly  to  Program  Managers,  consolidation  with  AEC  would  jeopardize 
the  independence  of  AEC  evaluations.  This  argument  is  easily  countered  by  the  fact  that  the 
AMSAA,  SLAD  and  HRED  analysts  that  currently  support  Program  Managers  are  typically  the 
same  analysts  that  support  AEC  evaluations,  so  evaluations  would  be  no  less  independent  after 
consolidation  than  they  are  today. 

JOINT  OPERATIONAL  TEST  REALISM 

Several  alternative  courses  of  action  (COA)  to  the  current  Operational  Test  process  could 
conceivably  enhance  the  operational  realism  in  the  representation  of  joint  operations  in  the 
events  that  are  used  in  the  Army’s  Operational  Evaluations. 

COA  0:  This  COA  is  the  status  quo,  in  which  ATEC,  the  Army's  Operational  Test  Agency 
executes  the  Operational  Test  using  existing  policy  and  procedures.  In  this  COA,  the  Army’s 
evaluator  determines  the  extent  to  which  Joint  “players”  are  necessary  in  the  Operational  Test 
and  is  responsible  for  ensuring  (through  existing  Service  Operational  Test  Agency  coordination 
channels)  their  participation. 

COA  1 :  In  this  COA,  the  Service  Operational  Test  Agencies  are  replaced  by  a  Joint 
Operational  Test  Agency,  which  is  responsible  for  the  planning  and  execution  of  the  Operational 
Tests  for  all  programs  across  DoD. 

COA  2:  In  this  COA,  a  dedicated  Joint  Transformation  Test  Unit  is  created,  this  unit 
conducts  all  Operational  Tests. 

COA  3:  In  this  COA,  traditional  Operational  Testing  is  bypassed,  with  the  system  being 
produced  and  fielded  upon  the  completion  of  Developmental  Testing.  Operational  evaluation  is 
conducted  post-deployment  based  on  a  Joint  National  T raining  Capability  or  Combatant 
Command  exercise.  In  recent  years,  significant  efforts  have  been  made  under  the  auspices  of 
the  Defense  Test  and  Training  Steering  Group  to  modernize  the  instrumentation  used  at  our 
nation's  training  ranges35,  making  their  data  collection  capabilities  similar  to  those  of  Operational 
Test  ranges.  Therefore,  it  is  feasible  that  exercise  venues  could  provide  the  data  elements 
required  by  operational  evaluators. 
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ANALYSIS 

Comparison  of  the  four  proposed  courses  of  action  is  based  upon  their  associated  risks  in 
the  dimensions  of  Test  Unit  Realism,  Test  Operations  Realism,  Operational  Effectiveness, 
System  Cost,  Force  Readiness,  Program  Schedule  and  Statutory  Compliance.  Table  1 
summarizes  the  comparison. 


COA  0 

Status  Quo 

COA  1 

Joint  OTA 

COA  2 

Dedicated  Unit 

COA  3 

Exercise 

Test  Unit 

Realism 

Medium  Risk 

Medium  Risk 

Low  Risk 

Low  Risk 

Test  Operations 

Realism 

Med/High  Risk 

Medium  Risk 

Low  Risk 

Low  Risk 

Operational 

Effectiveness 

Low  Risk 

Low  Risk 

Low  Risk 

High  Risk 

System  Cost 

Low  Risk 

Low  Risk 

Low  Risk 

High  Risk 

Force  Readiness 

Low  Risk 

Low  Risk 

Medium  Risk 

High  Risk 

Schedule 

Medium  Risk 

Medium  Risk 

Low  Risk 

High  Risk 

Statutory 

Low  Risk 

Medium  Risk 

Medium  Risk 

High  Risk 

TABLE  1.  COMPARISON  OF  COA  RISKS 


TEST  UNIT  REALISM 

Since  COA  3  involves  collecting  data  with  line  forces  in  Joint  exercises,  it  presents 
extremely  low  risks  in  test  unit  realism.  COA  2  also  has  low  risk  in  this  dimension,  as  long  as 
care  is  taken  to  ensure  that  the  dedicated  test  unit  does  not  evolve  into  a  “golden  unit” 
unrepresentative  of  the  rest  of  the  force.  The  medium  risks  associated  with  COA  0  and  COA  1 
are  due  to  the  difficulties  in  obtaining  commitments  for  realistically  sized  test  units. 

TEST  OPERATIONS  REALISM 

COA  2  and  COA  3  present  low  risk  in  this  dimension  because  both  would  employ  a  Joint 
unit  exercising  realistic  Joint  operations.  COA  2  poses  medium  risk  because  of  difficulties  in 
obtaining  commitments  for  realistically  sized  test  units,  and  COA  0  has  Medium/High  risk 
because  of  the  additional  challenges  that  an  Army  test  organization  has  in  securing  Joint 
players  in  operational  tests. 
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OPERATIONAL  EFFECTIVENESS 

COA  0,  COA  1 ,  and  COA  2  present  low  risk  in  this  dimension  because  Operational  Tests 
are  designed  to  stress  the  system-under-test  and  its  operators  over  a  wide  range  of  conditions, 
threats  and  environments.  Uncontrolled  variables  associated  with  the  “tree-play”  nature  of 
exercises,  and  the  requirement  to  not  interfere  with  training  objectives  would  likely  limit  the 
scope  of  conditions  and  threats  exercised  in  COA  3 36.  Therefore ,  there  is  high  risk  that  system 
operational  deficiencies  could  go  undetected  in  COA  3. 

SYSTEM  COST 

COA  0,  COA  1  and  COA  2  present  low  risk  in  this  dimension  because  “T&E,  especially 
OT&E  is  not  a  big  fraction  of  the  overall  budget  and  schedule...  Army  OT&E  costs  0.91%  of 
acquisition.’®7  COA  3  poses  high  cost  risks  because  of  the  likelihood  of  system  design  and 
performance  deficiencies  remaining  undetected  until  after  purchase  and  fielding,  when  costs  to 
fix,  retrofit,  and  confirm  are  much  higher. 

FORCE  READINESS 

COA  0  and  COA  1  pose  low  readiness  risk  because  Operational  Tests  typically  have 
minimal  impact  on  unit  rotations.  Since  the  formation  of  a  dedicated  test  unit  would  reduce  the 
units  and  equipment  available  for  operational  deployment,  COA  2  poses  a  medium  readiness 
risk.  COA  3  presents  a  high  readiness  risk  since  there  is  a  high  risk  of  system  deficiencies 
remaining  undetected  until  after  fielding  in  COA  3. 

SCHEDULE 

The  availability  of  a  dedicated  test  unit  in  COA  2  eliminates  test  schedule  risk  -  the  unit 
will  be  available  as  soon  as  the  equipment  is  ready  to  test.  Lacking  a  dedicated  unit,  when 
operational  tests  are  delayed  because  of  problems  in  development,  the  test  schedules  may 
need  to  be  further  delayed  to  synchronize  with  test  unit  rotations  -  therefore ,  COA  0  and  COA  1 
present  medium  schedule  risk.  There  has  been  a  recent  trend  to  cancel  major  exercises 
because  of  “real  world”  operational  commitments .  Therefore ,  COA  3,  which  relies  on  exercises 
for  operational  evaluation  data,  poses  a  high  schedule  risk. 

STATUTORY 

COA  0  presents  a  low  risk  in  this  dimension  because  the  current  Army  process  is  in  full 
compliance  with  existing  statutory  requirements.  COA  1  and  COA  2  pose  medium  risk  because 
they  reverse  the  synergistic  gains  that  were  realized  in  the  formation  of  ATEC  -  gains  which 
have  been  recognized  and  recommended  by  the  Defense  Science  Board  (DSB)  for  adoption  in 


15 


all  services38.  COA  3  poses  high  risk  in  this  domain  because  the  elimination  of  Operational 
Testing  from  the  defense  acquisition  process  would  require  the  elimination  of  Section  2399  of 
Title  1 0,  United  States  Code  -  an  extremely  unlikely  event. 

JOINT  OPERATIONAL  TEST  REALISM  RECOMMENDATION 

While  COA  3  provides  the  highest  promise  of  unit  and  operational  realism,  risks  in  other 
dimensions  make  COA  3  the  least  desirable  alternative.  If  improvements  in  unit  and  operational 
realism  are  truly  desired,  COA  2  (Dedicated  Unit)  should  be  pursued.  If,  however,  current  “real 
world”  operational  demands  on  force  structure  and  resources  prevent  the  acceptance  of  COA 
2's  higher  cost  and  readiness  risks,  then  COA  0  (Status  Quo)  should  be  maintained. 

CONCLUSIONS 

The  organizational  consolidation  of  SLAD,  HRED,  AMSAA  and  HRED  proposed  in  this 
report  will  provide  AEC  with  the  resources,  as  well  as  with  the  responsibility  to  conduct  all 
aspects  of  its  continuous  independent  evaluation  mission.  It  will  also  avail  AEC  of  the  resources 
and  responsibility  for  the  Army’s  contributions  to  the  Joint  T echnical  Coordinating  Group  for 
Munitions  Effectiveness.  These  changes  will  simplify  organizational  interrelationships  within  the 
Army’s  materiel  acquisition  community  and  will  improve  efficiency. 

Because  of  the  expense  of  large-scale  Operational  Tests  and  the  difficulties  in  securing 
multi-service  “player”  units,  realistic  Operational  Testing  in  a  Joint  Operations  context  will 
remain  problematic.  Therefore,  the  current  T&E  process  for  evaluating  systems  in  a  Joint 
operational  context  should  be  maintained.  If  operations  render  Joint  test  units  unavailable,  Joint 
interoperability  must  be  certified  prior  to  deployment  by  using  Developmental  Test  results,  and 
post-deployment  data  must  be  used  to  evaluate  Joint  operational  effectiveness. 
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